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SERVICE FOR LOAD TESTING 
A TRANSACTIONAL SERVER OVER THE INTERNET 

Fietdof the Invention 

5 The present invention relates to methods for testing the operation of web-based and other transactional 

servers. 

Background of the Invention 
A variety of commercially-available software tools exist for assisting companies in testing the performance 
and functionality of their transactional servers and associated applications prior to deployment (The term 
10 'transactional server* refers generally to a multi-user computer system such as a web sit* that provides interactive 
services to users over a computer network.) Examples of such tools include the LoadRurotei* WinRunner® and Astra 
QutckTest® products of Mercury Interactive Corporation, the assignee of the present application. 

Using these products, a user can record or otherwise create a test script which specifies a sequence of user 
interactions with the transactional server. The user may also optionally specify certain expected responses from the 
15 transactional server, winch may be added to the test script as verification points. For example, the user may record a 
session with a web-based travel reservation system during which the user searches for a particular flight and may 
then define one or more verification points to check for an expected flight number, departure time or ticket price. 

Test scripts generated through this process are. 'played* or 'executed* to simulate the actions of users - 
typically prior to deployment of the component being tested. During this process, the testing tool monitors the 
20 performance of the transactional server, including determining the passffafl status of any verification points. Muhipte 
test scripts may be replayed concurrently to simulate the load of a large number of users. Using an automation 
interface of the loadRunner product, it is possible to dispatch test scripts to remote computers for execution. 

The results of the test are typicaBy communicated to the user through a series of reports that are accessible 
through the user interface of the testing tooL The reports may contain, for example, graphs or charts of the observed 
25 response times for various types of transactions. Performance problems discovered through the testing process may 
be corrected by programmers or system administrators prior to deployment 

One type of tut that is commonly performed prior to the deployment of a new or modified transactional 
server is a load test A load test generally involves simulating the actions of relatively large numbers of users to 
determine how the transactional server wfll perform under heavy loads. One purpose of such a test is to determine the 
30 number of concurrent users the system can handle before performance becomes unacceptable - sometimes referred to 
as the "breaking point* The breaking point can be determined, for example, by scaling or ramping up the number of 
simulated users over tone white monitoring server response times. One testing tool that provides functionality for 
performing load tests is the above-mentioned LoadRurmer product of Mercury Interactive. 

Some companies abo offer hosted services for monitoring the post-deployment operation of web sites. 
35 Mercury Interactive, for example, provides a service known as Topaz ActiveWatch w which monitors a deployed web 
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site from various points of presence on the Internet on a 24-hour or other basis. The response times, transaction 
results, end other data collected through the monitoring processes are aggregated within a database for customer 
viewing, and are used to generate real-time notification messages when problems occur. These types of services 
generally do not provide toad testing capabilities. 

One problem companies commonly encounter is e need to load test a web site or other Internet-based system 
on very short notice. This problem commonly arises when a company is about to deploy its web site, or a new web 
site application, for the first time. For example, a company may wish to implement and launch its new ecommerce 
service quickly in order to be the first to market.. Other situations in which this problem commonly arises include the 
following: (1) when the company expects a sudden increase in traffic as the result of promotions, new product 
offerings, a newsworthy went, or some other type of event, and (2) when the company upgrades or modifies an 
existing software component of the transactional server. 

In these situations, a company may not have the time or other resources needed to set up, run and analyze 
the results of a load test prior to launch. For example, the company may not have the hardware resources needed to 
generate a sufficiently large load, or may not have the technical expertise needed to develop, run and analyze the load 
tests. The abcvfrmentioned tools and services do not provide an adequate solution to this problem. 

Summary 

The present invention overcomes the above and other problems by providing a hosted service for load testing 
a web site or other transactional server over the Internet or other public network. The service allows the owner or 
operator of a transactional server fcustomerl to outsource the task of load testing a transactional server to a load 
testing service provider. The service provider tests the transactional server remotely from a server farm, based on 
information remotely collected from the customer, without the need to visit the customer's facility. 

To use the service, the customer initially registers with the service provider - preferably through a web site 
of the service provider. As part of the registration process, the customer provides the information needed by the 
service provider to define load tests and generate associated scripts. This information may include, for example, an 
overview of the system architecture, anticipated traffic levels, and descriptions of the transactions to be tested. 

To enable the service provider to access the transactional server over the Internet the customer preferably 
sets up a substantially replicated version of the transactional saver using a staging server. The staged transactional 
server may be placed at an internet address that is not generally known to ordinary users (e.g. 
www3tage.abcstore.com), and/or may be placed behind a firewall to limit access. The use of a staging server allows 
the transactional server to be tested from a remote location prior to actual deployment, while allowing the customer 
to continue development using the non-staged version of the transactional server. To assist the customer in setting up 
the staged transactional server, the service provider's web site may include staging server software in downloadable 
form. 
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Using the transactional server and the information provided by the customer, the service provider uses 
conventional tools to remotely generate and ran load tests ol the transactional server. The load tests are preferably 
executed from e server farm that provides sufficient processing power to test multiple high-volume web sites 
concurrently. The results of the load test including recommendations for reducing bottlenecks end other performance 
problems, may be communicated to the customer through the service provider's web site, by telephone, or by another 
communications method. 

In ens embodiment, the service provider makes e collaboration tool available that allows the customer, or e 
consultant of the customer, to remotely participate in the process of generating and running load tests. 



An illustrative embodiment of the invention will now be described with reference to the drawings, in which: 
figure 1 illustrates the general architecture of a hosted load testing service according to the invention, and 
illustrates end example web site being tested by the service: and 

Figure 2 illustrates the general sequence of steps that are performed by the customer and the service 
15 provider to load teste web site prior to deployment. 

Detailed Description of an Illustrative Embodiment 
Throughout this description, it will be assumed that the transactional server being tested is a web-based 
system or 'web site" that is accessible via the Internet. It win be recognized, however, that the inventive methods 
can also be used to monitor other types of transactional servers, including those that use proprietary protocols or are 
eccessibte only to internal users of a particular organization. For example, the underlying methodology can also be 
used to monitor internal intranets, two-tier client/server systems. SAP RJ3 systems, end other types of distributed 
systems. Figure 1 illustrates the primary components of e system 30 that provides a hosted load testing service 
according to the invention. Figure 1 also illustrates an example web site 50 that may be remotely load tested by the 
25 service. For purposes of the following description, the term 'service provider* refers to the primary business entity 
that operates the load testing service, and the term -customer' refers to the primary business entity responsible for 
the operation of the web she SO. 

As depicted in Figure 1. the system 30 includes e service provider web site 32 that can be accessed by 
potential and existing customers of the service. The web site 32 includes a registration application 34 that provides 
30 functionality for entities te register to use the load testing service. Registration may alternatively be accomplished by 
telephone, email or other communications method. The web site 32 may also include a reports server 36 for allowing 
customers to review analysis reports and graphs generated during load test execution. In addition, the web site 32 
may provide a hosted collaboration tool 38 for allowing the customer or a consultant to actively participate in the load 
testing process. 
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As further depicted in Figure 1. the loed testing service is implemented using en errey or farm of servers 40 
that host a toad testing application 42. Each physical server 40A of the array is preferably capable of simulating the 
load of many thousands of concurrent users of the web she 50. The total processing power of the server farm 40 is 
pref erebfy sufficient to concurrently stress multiple high-traffic web sites to or beyond the breaking point so that 
multiple customers can be served simultaneously. The physical servers 40A are preferably interconnected by one or 
more local area networks (not shown) that connect the servers to a backbone of the Internet. In one implementation, 
the server farm includes 20 dual-processor. 500 MHz servers with 1 GB of memory ami a bandwidth of 46B/second. 
and can deliver a system loed equivalent to that of 100,000 concurrent users. Although all of the servers 40A are 
preferably located in a single location, some or aU of the servers 40A could be distributed geographically. 

The load testing application 42 is preferably the commertiaRy-available LoadRurmer® version 6.0 product of 
Mercury Interactive Corporation, although other load testing programs can be used. Using a controller component 
42A of the load testing application At a human operator can remotely specify the load testing operations to be 
performed by each server 40A. including, for example, the number of virtual users to be used on each server, the 
transactions to be executed by each such virtual user, the performance parameters to be monitored, etc. Methods for 
setting up and executing load tests are well kmrwn in the art. and therefore are not described in detail herein. The 
controller 42A may optionally implement a load balancing algorithm which assigns load testing tasks (Vuser execution 
events, etcj to individual load testing servers 40A so as to evenly distribute the processing load. The controller 42A 
may run on a separate computer that is remote from the server farm 40. such as at a remote development facility of 
the service provider. 

The web site 32 and the load testing servers 40A access a database 44 of customer data. The information 
stored within this database 44 may include, for example, customer information provided during registratioa test 
scripts and load testing scenarios developed by the service provider, raw performance data generated during test 
execution, end annotated performance charts and graphs prepared by the service provider for customer viewing. The 
database 44 may be implemented as a file server, a single database system, a set of distinct database systems, or 
enother data storage scheme, and is shown as a single database largely for illustrative purposes. 

As further illustrated in figure 1. the customer web site 50 is typically implemented physical web servers 
52. application servers 53. database servers 54 and back-end databases 56. For load testing purposes, the web she 
50 is preferably made accessible to the service provider using a staging server. Staging server software that can be 
used for this purpose is available from such companies as Netscape Corporation and Coast Software. The staging 
software may optionally be provided to customers in downloadable form on the service provider's web she 30. 

The staged web site 50 is preferably a substantially replicated or mirrored version of the actual web she; or 
at least of the web site functionality to be tested. The same or a different set of physical servers 52 54 may be used 
to implement the staged and the actual implementations of the web site 5a It is desirable, however, that the 
processing power of the staged web she be approximately the same as that of the actual or anticipated web she. The 
staged web she 50 may be placed at a special Internet address (eg., www.stegeo.bcstomxom) that is not generally 
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known to users, and/or may be placed behind a firewall. One benefit of using a staging server for load testing is that 
the web site's components ere generally isolated from damage during load testing. Another benefit is that the 
customer can continue development of the web site during load testing. 

Figure 2 illustrates a preferred business process used to implement the hosted service, with actions 
performed by the customer shown in the left-hand column and actions performed by the service provider shown in the 
right-hand column. The illustrated actions of the service provider ere preferably performed by a team of load testing 
experts. Some or aO of the illustrated communications between the customer and the service provider ere preferably 
in the form of secure postings to the service provider's web site 32. 

In step 1 of Figure 2. the customer registers with the service provider. During registration, the customer may 
provide such information as the company name; payment information, a description of the web site end the associated 
transactions to be tested [including relevant URLs, username/password pairs, data values, etc.), a description of the 
web site's architecture, expected traffic levels, the expected launch date, details of any related Service level 
Agreements ISlAs) with ISPs or others, end the class of load testing service desired (e.g., load testing only, load 
testing and functionality testing, eta Some or all of this information is preferably obtained through registration 
forms on the service provider's web site 32. The customer also preferably specifies a username and password for 
accessing a private area of the service provider's web site 32 that b used for communications and possibly 
collaboration between the customer and the service provider. As mentioned above, the information submitted by the 
customer is stored in the customer database 44. 

The customer may also be required to pay some or all of the service provider's fees during the registration 
process, such 35 by entering a credit card number. The fees charged by the service provider may, for example, be 
based on one or more of the following example criteria, and may be calculated automatically based on the information 
provided by the customer: number of days until launch date, number of transactions to be tested, expected level of 
traffic and whether or not web site functionality (content of server responses) will be tested. If the customer does 
not already have a staging server, the fees may also include e licensing fee for use of staging server software provided 
by the service provider. 

As indicated by step 2a, the service provider uses the information provided by the customer to define testing 
goals and to design the load tests. The load testing designs and goals am preferably presented to the customer for 
approval, as depicted by step 2b. The customer may also make the staging site accessible to the service provider et 
this time. 

As depicted by step 3a, the service provider then develops the load tests to be run against the staged she. 
The load test scripts are preferably recorded using a recorder component of LoadRurmer 6.0, but could alternatively be 
generated manually or using other tools. Each script may represent a particular transaction or sequence of 
transactions to be performed by a virtual user. For example, a script for use with a merchant's web site may start out 
with an access to a search page, followed by a submission of a search query, followed by a selection of a search 
resuh hem for viewinftfc^owed by a purchase of tte If ^ functionality is to be verified during had testing. 
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the service provider may insert verification points that specify expected server responses, such as an expected order 
confirmation message from the transactional server. The service provider may also parameterize variables so that 
different data sets can be used on different iterations of the test script. Preferred methods for recording 
transactions, adding verification points and parameterizing variables are described in U.S. appL no. 09/337,082, filed 
June 21, 1999. The service provider also executes and debugs the scripts. The script files, and the associated 
scenario files that specify how the scripts will be used to test the web site 50, are stored in the customer database 
44 for subsequent use. As depicted by step 3b, the customer may approve the scripted transactions prior to actual 
testing. 

In step 4a. the service provider then runs the load test against the staged web site 50 from tha array of 
savers 40. The service provider preferably starts out with a relatively small number of virtual users and then ramps 
up this number over time. IP spoofing is preferably used to create a traffic pattern in which different simulated users 
have different IP addresses. During load test execution, the real-time monitors of the load testing application 42 
isolate performance bottlenecks by splitting end-to-end transaction response times into separate components 
associated with the dient, the network, and the servers. For example, a server monitor component (not shown) 
locates any problems associated with the web site's web servers 52, anpBcation servers 53, and database servers 54. 
Smilarfy, a network delay monitor (not shown) isolates network performance problems by breaking down the network 
topology between the cfiem and the server and measuring tha network delay along each segment. 

Tha transaction response times and other performance data generated during load testing are aggregated 
within the customer database 44, and are reviewed and analyzed by the service provider using the various charts and 
reports provided by the load testing application 42. Some or all of the performance data may also be made available 
to tha customer for viewing via the service provider's web site 32. In addition, as mentioned above, the customer or a 
consultant may be able to participate in the toad testing process, such as by recording additional transactions or 
def bung new execution scenarios, using a hosted collaboration tool 38 on the service provider's web site 32. 

The results of the service provider's analysis may be communicated to the customer through the service 
provider's web site 32 (e*, through annotated performance graphs and charts), by telephone, and/or by other 
communications method. As part of this process, the service provider will typically, suggest modifications that wfll 
improve the performance of the web site 50. For example, the service provider might inform the customer that the 
web site's database servers 54 take too long to lock once the load reaches a certain level or that the customer's ISP 
is violating a Service level Agreement by providing insufficient throughput As depicted by steps 4b and 5, once the 
customer makes any suggested changes to the stage) web site 50, the service provider wiO typically re-ran the load 
tests to evaluate the effects of the changes. Once load testing is complete, the service provider may make tha test 
scripts and associated files avaflabte to the customer to use for post-deployment testing or monitoring of the web site 
50. 

Although the invention has been described in terms of one particular embodiment other embodiments that 
are apparent to those of ordinary skin in the art including embodiments which do not provide al) of the features and 
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advantages set forth herein, are also within the scope of this invention. Accordingly, the scope of the invention is 
defined by the claims that follow. 
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WHAT IS CLAIMED IS: 

1. A method of toad testing a transactional server, comprising: 
remotely obtaining information about the transactional server from a business entity associated 

with the transactional server; 

generating a load test based on the information remotely obtained from the business entity, 
wherein the load test specifies actions of simulated users of the transactional server; and 

running the load test from a server farm that is remotely coupled to the transactional server by a 
public network to apply a load to the transactional server over the public network. 

2. The method as in Claim 1. wherein running the load test comprises rurmmg the load test against a 
staged implementation of the transactional server. 

3. . The method as in Claim 1, wherein the public network comprises the Internet 

15 4 - nwthod 83 < n Cte ™ wherein remotely obtaining information about the transactional server 

comprises using a web site to register the business entity. 

5. The method as in Claim 4. further comprising posting results of the load test on the web site for 
remote viewing by the business entity. 
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6. The method as in Claim 1, wherein running the load test comprises applying a load to a version of 
the transactional server that has not yet been deployed. 



7. The method as in Claim 1, wherein the transactional server is a web site system. 

25 

8. A method of testing a transactional server, comprising: 

providing a staged implementation of the transactional server that is remotely accessible via the 
Internet and 

running a load test against the staged implementation of the transactional server from a server 
30 ferm that b remote * rom the transactional server and coupled to the transactional server by the Internet. 

9. The method as in Claim 8, further comprising developing the toad test based on information 
remotely obtained from an operator of the transactional server through an online registration process. 
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10* The method as in Claim 9, further comprising posting results of the toad test on a web site for 
remote viewing by the operator. 

11. The method as in Claim 8, wherein running the load test comprises running the test against a 
5 version of the transactional server that has not yet been deployed. 

12. The method as in Claim 8, wherein providing a staged implementation of the transactional server 
comprises placing the staged implementation at an Internet address that is not generally known. 

10 13. The method asm Claim 8. wherein the transactional server is a web site system. 

14. A system for load testing a transactional server over the Internet, comprising: 

a plurality of servers that are configured to load test transactional servers over the Internet from a 
remote location; and 

15 a web site that is configured to collect information from operators of the transactional servers to 

be load tested, including information used by a service provider to develop load tests to be executed by the 
plurality of servers. * 

15. The system as in Claim 14, wherein the web site hosts a reports server application for allowing 
20 the operators to view results of the bad tests. 

IB. The system as in Claim 14, wherein the web site hosts a collaboration application that allows an 
operator to remotely participate in the development of load tests. 



25 



17. The system as in Claim 14, further comprising a controller component that is configured to control 
the plurality of servers from a remote location. 



30 
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